Fix chat-to-proposal NLP gap: natural language now produces proposals#602
Fix chat-to-proposal NLP gap: natural language now produces proposals#602Chris0Jeky merged 6 commits intomainfrom
Conversation
…arsing gap When the intent classifier detects actionable intent in natural language (e.g., "create new onboarding tasks"), this extractor produces structured instructions (e.g., create card "Onboarding tasks") that the regex-based parser can consume. This is the core fix for the #570 NLP gap.
…language The Mock provider now uses NaturalLanguageInstructionExtractor when the classifier detects actionable intent, producing Instructions that ChatService can pass to the parser instead of the raw user message. Fixes the core #570 scenario where natural language fails to produce proposals.
When OpenAI structured JSON parsing fails and the provider falls back to the static classifier, also extract structured instructions so natural language can still produce parseable instructions for the planner.
When Gemini structured JSON parsing fails and the provider falls back to the static classifier, also extract structured instructions so natural language can still produce parseable instructions for the planner.
Cover card create (quoted titles, natural language, various verbs), card move/archive/update with IDs, board rename, edge cases (null, empty, unknown intent), and title cleaning logic.
Adversarial Self-ReviewIssues Found and FixedBug: Remaining Observations (Non-Blocking)
Action Items
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the NaturalLanguageInstructionExtractor service to convert natural language user messages into structured instructions compatible with the AutomationPlannerService. This extraction logic is integrated into the Gemini, OpenAI, and Mock LLM providers as a fallback mechanism when structured JSON parsing fails. Key feedback includes correcting a logic error where board creation incorrectly generated card creation instructions and utilizing a defined but unused regex pattern for unquoted column names in move operations.
| { | ||
| var name = quotedMatch.Groups[1].Value.Trim(); | ||
| if (!string.IsNullOrWhiteSpace(name)) | ||
| return new List<string> { $"create card \"{name}\"" }; |
There was a problem hiding this comment.
The ExtractBoardCreateInstructions method incorrectly returns a create card instruction when the intent is board.create. Since the AutomationPlannerService does not currently support a create board command, this method should return an empty list to avoid creating unintended cards.
return new List<string>();| // Try quoted column name first | ||
| var columnMatch = ColumnNamePattern.Match(message); | ||
| if (columnMatch.Success) | ||
| { | ||
| var columnName = columnMatch.Groups[1].Value.Trim(); | ||
| return new List<string> { $"move card {cardId} to column \"{columnName}\"" }; | ||
| } | ||
|
|
||
| return new List<string>(); |
There was a problem hiding this comment.
The ColumnNameUnquotedPattern regex is defined but not used. It should be integrated here as a fallback to support move instructions where the column name is not quoted (e.g., "move card {id} to column Done").
// Try quoted column name first
var columnMatch = ColumnNamePattern.Match(message);
if (columnMatch.Success)
{
var columnName = columnMatch.Groups[1].Value.Trim();
return new List<string> { $"move card {cardId} to column \"{columnName}\"" };
}
// Fallback to unquoted column name
var unquotedMatch = ColumnNameUnquotedPattern.Match(message);
if (unquotedMatch.Success)
{
var columnName = unquotedMatch.Groups[1].Value.Trim();
return new List<string> { $"move card {cardId} to column \"{columnName}\"" };
}
return new List<string>();ExtractBoardCreateInstructions incorrectly emitted create card instead of returning empty (AutomationPlannerService has no board create parser pattern). Also remove unused ColumnNameUnquotedPattern field.
Self-Review Fixes Applied (commit 0087725)
All 1,737 backend tests pass after fixes. |
Adversarial Code Review -- PR #6021. [High] Injection via unquoted titles containing quote charactersFile: When the Example input:
This is a data corruption bug -- titles are silently mangled for any input containing quotes in the natural-language path. Recommendation: Strip or escape embedded quote characters in 2. [High]
|
| # | Severity | Issue |
|---|---|---|
| 1 | High | Embedded quotes in NL-extracted titles corrupt the instruction format |
| 2 | High | CleanExtractedTitle inline regexes lack timeout (inconsistency) |
| 3 | Medium | column.reorder intent not handled -- same bug class as #570 |
| 4 | Medium | board.create returns empty -- misleading promise with no proposal |
| 5 | Medium | CardUpdateInstructions positional ambiguity |
| 6 | Low | No integration test for full ChatService pipeline |
| 7 | Low | No adversarial input tests |
| 8 | Low | Multi-card intent collapsed to single card |
Items 1-4 should be addressed before merge. Items 5-8 are acceptable as follow-up.
1 participant
Summary
Closes #570. Natural language chat messages that express actionable intent (e.g., "create new onboarding tasks for people who aren't technical") now produce proposals instead of failing with parse errors.
NaturalLanguageInstructionExtractor: Bridges the gap between intent classification (which detects that a message is actionable) and instruction parsing (which requires structured syntax likecreate card "title"). When the classifier detects actionable intent, the extractor translates the natural language into structured instructions the regex parser can consume.Instructionswhen the classifier detects actionable intent, so ChatService passes structured instructions to the planner instead of raw natural language.What Changed
NaturalLanguageInstructionExtractor.csMockLlmProvider.csInstructionsfor actionable messagesOpenAiLlmProvider.csGeminiLlmProvider.csNaturalLanguageInstructionExtractorTests.csTest plan
NaturalLanguageInstructionExtractor(quoted titles, natural language, card operations, board operations, edge cases)